Conjunctive Filter: Breaking the Entropy Barrier

We consider a problem for storing a map that associates a key with a set of values. To store n values from the universe of size m, it requires log2 ( m n ) bits of space, which can be approximated as (1.44 + n) log2 m/n bits when n ≪ m. If we allow ε fraction of errors in outputs, we can store it with roughly n log2 1 ε bits, which matches the entropy bound. Bloom filter is a wellknown example for such data structures. Our objective is to break this entropy bound and construct more space-efficient data structures. In this paper, we propose a novel data structure called a conjunctive filter, which supports conjunctive queries on k distinct keys for fixed k. Although a conjunctive filter cannot return the set of values itself associated with a queried key, it can perform conjunctive queries with O(1/ √ m) fraction of errors. Also, the consumed space is nk log2 m bits and it is significantly smaller than the entropy bound n2 log2 m when k ≥ 3. We will show that many problems can be solved by using a conjunctive filter such as full-text search and database join queries. Also, we conducted experiments using a real-world data set, and show that a conjunctive filter answers conjunctive queries almost correctly using about 1/2 ∼ 1/4 space as the entropy bound.